Welcome back to deep learning. So today we want to discuss in the last part about weekly and self-supervised learning a couple of new losses that can also help us with the self-supervision.

So let's see what we have on our slides. Today part number four and the idea is contrastive self-supervised learning.

So in contrastive learning you try to formulate the learning problem as a kind of matching approach.

So here we have an example from supervised learning and the idea is then to match the correct animal with respect to other animals.

So the learning task is whether the animal is the same or a different one.

And this is actually a very powerful of training because you can avoid a couple of disadvantages in generative or context models.

So for example pixel level losses could overly focus on pixel based details and pixel based objectives often assume pixel independence.

And this reduces the ability to model correlations or complex structures.

So here we essentially can then build abstract models that are also built in a kind of hierarchical way.

Now we have this supervised example but of course this also works with many of the different pseudo labels that we've seen earlier.

So we can then build this contrastive loss and all that we need is the current sample X, some positive sample X plus and then negative samples that are all from a different class.

In addition we need a similarity function S and this could for example be a cosine similarity.

You could also have a trainable similarity function but generally you can also use some of the standard similarity functions that we already discussed in this lecture.

Furthermore you want to apply then your network F on X and compute the similarity and the goal is then to have a higher similarity between the positive sample and the sample under consideration and all the negative ones.

This then leads to the contrastive loss. Sometimes this is also called the info normalized cross entropy loss and there's also other names in the literature.

The end payer loss, the consistency loss, the ranking based NCE and so on.

It's essentially a cross entropy loss for an N-way softmax classifier and the idea here is that you then have essentially the positive examples.

And this is then normalized by all of the examples here.

I'm splitting the two still in the first time but you can then simply see that the split is actually a sum of all of the samples and this then yields a kind of softmax classifier.

So minimizing the contrastive loss actually maximizes a lower bound on the mutual information between F and F plus as shown in those two references here.

And there's also of course a common variation that introduces a temperature hyperparameter that is shown in this example so you divide by an additional tau.

The contrastive losses are very effective and they have two very useful properties.

On the one hand they align so similar samples have similar features and on the other way they create uniformity because they preserve the maximal information.

Let's look into an example how this can be constructed and this is reference 31.

So let's say you start with a mini-badge of N samples.

Then each sample is applied with two different data augmentation operations.

Then this leads to two N augmented samples.

So for every sample in the batch you get two different augmentations.

Here we take T and T prime that we apply to the same sample and you know that this sample is then the same.

And your transformations T and T prime are taken from a set of augmentations T.

Now you end up with one positive pair for each sample and two times N minus one negative pairs because they're all different samples.

And then you compute a representation through the base encoder F that then produces some H which is the actual feature in presentation that we're interested in.

And an example for F could be like a ResNet-50.

Then on top of this we have a representation head G that then does an additional dimensionality reduction.

And note that both G and F are the same in both branches.

So you could argue this approach has considerable similarities to what is called a Siamese network.

So you then obtain two different Zs, Zi and Zj from G. And with those Zs you can compute your contrastive loss that is then expressed as the similarity between the two Zs over the temperature parameter tau here.

And you see this is essentially the same contrastive loss that we already seen earlier.

Contrastive losses of course can also be combined in a supervised way and this then leads to supervised contrastive learning.

And here the idea is that if you just perform the self-supervised contrastive you have positive examples versus the negative ones.

And with the supervised contrastive loss you can then also embed additional class information.

So this has additional positive effects.

So let's see how this works. So let's summarize a bit what the difference between supervised learning, contrastive learning and supervised contrastive learning.

Well in supervised learning you would essentially have your encoder that is shown here as this trapeze.

Then you end up with some description that is 2048 dimensional vector and you then train a classification head that produces the class dog using the classical cross entropy loss.

Now in the contrastive learning you would then expand on this. So you would have essentially two coupled networks and you have the two patches that are either the same or not the same with a different augmentation technique.

And you train these coupled weight shared networks, produce 2048 dimensional representations and on top you have this representation head and you use the contrastive loss on this representation head, not directly on the representation layer itself.

Now if you combine the two into supervised contrastive you essentially have the same setup as in the contrastive loss but on the representation layer you can then augment an additional loss that works strictly supervised.

And there you still have the typical softmax that goes to let's say a thousand classes and predicts dog in this example and you couple it on the representation layer to be able to fuse contrastive and supervised losses.

So the self supervised has no knowledge about the class labels and it only knows about one positive example. The supervised has knowledge about all the class labels and has many positives for example.